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ABSTRACT 

The formative evaluation of a new Malaysian secondary 
school language curriculum is described.. Reasons for choosing 
formative over summative evaluation are examined, and implementation 
of the evaluation to date is detailed. In the first two phases, state 
education officials were trained in principles and procedures of 
program evaluation, including all stages from planning through 
reporting of results. These phases included seminars, site visits, 
and refinement of the overall evaluation plan. The three subsequent 
phases of the evaluation are to include: (l) another seminar to 
define communication channels, introduce more sophisticated 
analytical and reporting procedures, and begin national level 
studies; (2) review progress, expand use of formative evaluation 
throughout the country, and provide consultation in data collection 
to the states; and (3) preparation of the final report and 
recommendations. Topics identified by the different states for 
investigation in the evaluation project are noted. The most common 
are aspects of classroom management, teaching methodology, and 
learning. State officers' perceptions of what they have learned about 
evaluation and about the curriculum's implementation in the project's 
first two phases are summarized. (MSE) 
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NATIONAL LEVEL FORMATIVE EVALUATION: 
SOME FIRST STEPS 



AU Abdul Ghmi and Bnm Hunt 



Histor;' and Background 
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From the begjbmng of the academic year in January 1987 a new integrated language 
curriculum: Kurikulum Bersepadu Sekolah Menengah (KBSM), was introduced into 
Malaysian secondary schools. This was for all the languages taught in Malaysia at secondary 
school level: Bahasa Malaysia, English, Chinese, Tamil. The KBSM was to be introduced 
progressively year by year beginning with Form I in 1988. 1990 sees its introduction into 
Form in. The introduction of the KBSM curriculum for other subjects is following one year 
behind. 

The KBSM curriculum for English aims to integrate the language skills (Ustening, 
speaking, reading and writing); the language areas of lexis, phonology and grammar as well 
as knowledge of other subjects on the timetable and good moral values as indicated by the 
National Education Philosophy of Malaysia. 

The Ministry of Education felt that the introduction of the KBSM curriculum should 
be monitored in order to gauge its effects on teaching and learning in the classroom. It 
was felt that teachers and students, as the ultimate users of the new curriculum, would have 
most to say about its effectiveness. 

There are three reasons for this dedsioa. 

Firstly, it is hoped to obtain a clear picture of the effects of the new curriculum on 
classroom teaching and learning. 

Secondly, it is hoped that teacher-generated ideas for curriculum development will be 
more sensitive to learners' needs. 

Thirdly, modifications to the syllabus are likely to be longer lasting if they are 
recommended and carried out from 'bottom up' rather than prescribed from 'top down'. 

Having decided, in principle, to monitor the introduction of the new curriculum a 
choice had to be made between a summative evaluation and a formation evaluation. 

Several reasons decided in favour of a formative, rather than a summative, 
evaluation. 

Firstly, as the KBSM curriculum is to be introduced progressively year-by-year 
beginning with the fijst year and ending with fifth year of secondary schooling, conducting a 
summative evaluation exercise at the end of five years would be too late to be of optimum 
benefit. 

Secondly, in addition to being too late, it was felt that a summative evaluation would 
be less subtle and sensitive for the task of monitoring the KBSM programme; and that the 
ultimate aim of the formative evaluation exercise should be the creation of a delicate set of 
evaluation instruments. 
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Dermot Murphy explicitly deals vnth this aspect: 

'Summative evaluation, most often realized as assessment of learner performance, can 
produce results open to interpretation as dealing with something fixed: what the 
results should be. Poor results are due to the learners not achieving the objectives of 
the course. Its focus is limited, and the assessment m^^ not ff)^ any due to what needs 
adjusting to match leamer achievement to cumadum expectation. The evaluation does 
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not produce enough information and reinforces professional secrecy and 
entreachment*. (Our emphasis) 

Dermot Murphy 'Evaluation in Language 
Teaching: Assessment, Accountability and 
Awareness*. Lancaster Practical Papers in 
English Language Education. Volume 6. 
Pergamon Press (1985) 

Thirdly, the Ministry is eager to encourage teachers to contribute their ideas to 
curriculum evaluation and development, and by initiating a *bottom-up* formative evaluation 
project it is hoped to engage in valuable and informative dialogues between teachers and the 
Ministry of Education. 

Fourthly, it is felt that by encouraging classroom-based investigations of the 
curriculum in use by teachers and learners, teachers can develop their own self awareness of 
their classroom techniques and methodology with a view to professional development in both 
the long and short terms. 

Another important consideration relates to the separate responsibilities of the two 
(Uvisions of the Ministry whose work relates to schools and curricula. Schools Division has 
the responsibility, through State Education Offices, for the administration and management 
of schools,, teachers and students. The Curriculum Development Centre has the 
responsibility for curriculum policy, planning, the introduction of new curricula and related 
training programmes for Resource Persons. It is important that any programme of 
monitoring the KBSM does not duplicate the work of other ministerial divisions. Diverse 
divisions of the ministry are, however, kept informed of English language programmes via 
periodic meetings of inter-divisional committees. The progress and interim findings of the 
formative evaluation programme are thus reported regularly. 

We hope that the results of a formative evaluation will, 

i pro\dde precise & accurate information to relevant divisions of the Ministry about 
the existing situation of teaching and learning in Malaysian secondary schools. 

ii allow refinements to be made to the curriculum in the light of experience. 

iii help in the planning of teacher education courses related to the KBSM 
curriculum. 



2 Implementation of the Programme 

The former English Language Adviser to Schools Division, Mr. Lionel Thompson, 
was asked to investigate the possibility of an evaluation project. He prepared a working 
paper in which he outlined areas for investigation. These included suggested 
investigation of systemic, environmental and pedagogic factors. An initial approach was 
made on paper to Mr. Dermot Murphy,lecturer in ELT at St. Mary* College, Twickenham, 
whose response was enthusiastic. 

To initiate the formative evaluation project Dermot Murphy was invited to Malaysia 
to conduct a one-week introductory training seminar, and to follow up with practical field 
training lasting a further week. 

Forty-four participants consisting of State Education Language Officers, Key 
Personnel and Resource Personnel from each of the fourteen states; officers from various 
interested divisions of the Ministry of Education (including Heads of Bahasa Malaysia, 
Chinese and Tamil), Curriculum Development Centre, Federal Inspectorate, Teacher 
Education Division, and Examinations Syndicate attended the seminar held in Melaka. 
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The seminar introduced participants to the principles and purpose of evaluation; 
information gathering techniques and sampling. A range of evaluation questionnaires and 
pro-forma were analysed. Participants discussed areas for evaluation, and identified a range 
of priorities. The areas for investigation suggested by Mr. Thompson were offered as a 
starting point. 

The participants prepared evaluation instruments, presented their ideas to fellow- 
participants and refined them in group discussion. As an aid to classroom observation 
techniques a video from Teaching and Learning in Focus* was presented, and discussed. A 
simulation was prepared and carried out so that participants could, in turn, present the 
notion of formative evaluation to teachers. Finally, participants organized and planned future 
work areas based on ideas gleaned from the seminar. 

During the practical field visits to Kelantan (the state in the extreme north-east of 
Peninsula Malaysia) and Melaka (the state in the south-west of Peninsula Malaysia), State 
Education Officers, in consultation with Dermot Murphy and school teachers, agreed on 
their priorities for investigation. 

In Kelantan these were: 

a investigate students' generally low motivation, particularly in rural schools, 
focussing on aspects of their attitudes towards English; knowledge of the 
English-speaking world; the place of English as an international language; and 
of proficiency in English as a requirement for employment; 

b try out and evaluate teaching techniques to use with slow learners; 

c assess teachers* understanding and use of methodology required by, or 
appropriate to, changes introduced by KBSM (e.g. pair and group work; 
integration of skills; integration of moral \ alues); 

d evaluate locally produced materials; 

In Melaka these were: 

a investigate motivation (as in Kelantan); examine in particular the lack of success 
a particular semi-rural school has had with the English Language Reading 
Programme; 

b assess the adequacy of the briefing teachers have been given on KBSM; 

c assess teachers' understanding and use of methodology required by or appropriate 
to changes introduced by KBSM (e.g. pair and group work; integration of skills; 
integration moral values). 

It was proposed at the Melaka seminar that participants would organize their own 
exploratory investigations and that a future seminar to allow reporting back would be 
arranged. Accordingly Mr. Dermot Murphy was reinvited to conduct a seminar in Penang in 
September 1989. 

Forty-four participants attended the Penang seminar. Participants were State 
Language Officers, Supervisors, Key Personnel, Resource Personnel, Inspectors and one 
teacher. This number included twenty-four officers who had attended the Melaka seminar 
who were directly involved in formative evaluation work, plus colleagues who had been 
introduced to the formative evaluation programme following the Melaka seminar. 

The aims of the seminar were to 

i report on evaluation carried out thus far in the individual States 

ii review principles and techniques for evaluation 

iii focus on the stages of analysis, interpretation and reporting of evaluation findings 

iv plan for the third stage of the project. 
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Participants reported back on their work to date and explained how their investigative 
instruments had been designed. Many participants felt that they had taken on too much 
worL 

Practical work on analysis and reporting was carried out from participants' work. 
Dermot Murphy reviewed a range of data gathering techniques: interviewing, observation, 
diary studies and case-study, to allow participants opportunities to 'triangulate* from their 
existing data. 

By the end of the seminar participants were able to focus their future work. 
Suggested areas were 

i teacher attitude to KBSM 

ii teaching techniques in KBSM classes 

iii pupil behaviour in KBSR (primary) and KBSM (secondary) classes. 



The Formative Evaluation Project in Operation 



There are several strands of personnel in our formative evaluation project at the 
Ministry of Education in Malaysia. These are: 



Personnel 

Mr Dermot Murphy 
St. Mary's College 
Twickenham 

Ministry of Education 
Officers 



Roles 

act as consultant to the 
programme 

provide academic input and practical training 

(through seminars). 

manage overall project 

provide support and guidance 

keep states up-to-date and in touch with each 

other. 



State Language Officers 
from all 14 states 

Key Personnel and Resource 
Persons &om all states 

Teachers in various schools 

Students in various schools 



design evaluation instruments 
collect data. 



) provide data and feedback from 
) personal experience. 

) 



From the inception of his association with our formative evaluation project Mr. 
Dermot Murphy has had three main tasks. 

Firstly he was to provide information about formative evaluation to officers involved 
in the project. Secondly to give training in preparing evaluation instruments, data collection, 
data interpretation and reporting. He did all this and more during our first two seminars 
and will continue with this work during our forthcoming seminar in July, 1990. 

Thirdly he was to make recommendations to the Ministry of Education concerning 
the development of the project in the longer term. His reports have been most useful and 
our project continues to benefit greatly from his advice. We are working closely with Mr, 
Murphy in respect of the stages in the project and in the direction that the project should 
take. 

The task of the Schools Division of the Ministry of Education is to oversee the 
development of the project from its inception to becoming a part of our educational 
administrative system. 
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We also have the responsibility to give support to the teams working on formative 
evaluation at state level through: 

- a quarterly newsletter (to keep states in touch and up-to-date) 

- visit to states (to give assistance and advise where needed) 

- organizing regular (usually twice a year) meetings with all state language officers 
(to review progress). 

Following Mr. Murphy's recommendations we have divided our formative evaluation 
project into five phases lasting over some 2 1/2 years from January 1989 until June 1991. 

In general, these phases cover the stages of development of the project from the 
initial introduction to the project in January 1989 until the submission of the first set of 
interpreted data along with a report of related recommendations to senior Ministry of 
Education officials m May 1991. The time framework is to a certain extent prescribed by the 
support and fimding by the Overseas Development Agency, London of an ELT adviser to the 
Schools Division of the Ministry of Education. 

(Details of our project stages are in appendix I). 



Areas of Investigation 

Within broad guidelines (e.g. the desirability of teacher-led, classroom-based 
research; the need to monitor the effectiveness of the KBSM curriculum) State Language 
Officers and their teams were purposely allowed a free hand in deciding their topics for 
investigation, (m collaboration with teachers), designing their investigative instruments 
and for drawing up their own administrative schedules. Although there is predictably a 
degree of overlap m the areas chosen for investigation, there are a range of differences as 
well as variety of mvestigative techniques. 

The most common topic is some aspect of teachers* classroom management and 
methodology with particular reference to integration of skills and content, small group 
activities, pairwork, use of teachmg materials and other teaching aids. 

Ten out of the fourteen states are investigating one or more aspects of classroom 
management and methodology. One in-depth study is concentrating on the classroom 
methodologies of teachers who have little or no ELT training but who have been seconded 
to teach ELT because of their English language proficiency. This investigation is using 
questionnaires and interviews, lesson observation, self-evaluation forms and diary studies. 

A smaller number of states (four in all) have chosen to direct their investigations 
towards learning. Specific areas of research are class participation, student interaction, 
learning styles, story telling and language games, and students' responses to different types of 
homework assignment. This research is being carried out through the use of questionnaires 
and interviews (with both teachers and jitudents). 

What will prove useful for our national level investigations is that different 
investigative tools have been chosen by the various states. This means that classroom 
management and methodology is being investigated using questionnaires and interviews 
(with both teachers and students); lesson observations, teacher self-assessment forms anr? 
teachers end-of-class checklists. As Dermol Murphy has repeatedly stressed the notion of 
triangulation during each of the seminars held so far, the fact that we will be able to seiect 
from a range of data collection techniques for our national level investigations will allow us 
to scrutinize our data gathering carefully and be more confident in the validity of our results. 

(See Appendices II and III for a fuller outline of areas of investigation, and Appendix IV for 
an outline of projects state by state). 
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What Have Wc I^^arnr d So Far? 



While preparing our paper for this conference we held a two-day meeting with our 
State Language Officers during which we asked them to give us an up-date of their projects 
with particular reference to any advice they could now offer, from personal experience, to 
those contemplating formative evaluation research projects. 

This is the advice they offer, more or less unaxumously. 

plan carefully 

prepare a ppropriate instruments 
get a third opinion on your instruments 
conduct a small pilot study to ensure validity of 
instnunents and whether users can imderstand 
it. 

try not to do too much! 

• investigate one aspect at a time 

• identify one area for analysis and investigate this 
thoroughly (NB triangulation) 

- start small to begin with. 

- precise, clear, with specific objective 

- students* questionnaires should be bilingual 
(Bahasa Malaysia & English) 

- do not involve too many people initially 
get a good team! 

get people who are willing to carry out their tasks. 

- spend time and effor; refining your system of 
reporting (it will save you both time and effort 
later!). 

Public Relations - try and involve other subjects and other 

interested parties publicize your efforts. 



Participants * responses 

At the Ministry we were concerned that the formative evaluation project would 
stimulate our personnel to initiate research projects. We were particularly anxious that State 
Language Officers and their teams, and teachers who were mvolvt^d in the project would not 
be so pressured by other priorities that they would be unwilling to conduct time-consummg 
investigations of the type we envisaged. 

After the introductory seminar m Melaka m January 1989 we learnt from informal 
feedback that a number of participants had reservations about the utility of the project or 
mdeed about the Ministry's sincerity in engaging in a dialogue with classroom practitioners. 
A proportion felt that the project would be short-lived. 

On the practical side participants felt unsure and perhaps over-cautions about what to 
do next. Sofr.e felt that they had not been given enough support during the seminar 
(although this was one aim of the 'discovery* learning approach). 

Nevertheless, state level teams suspended their disbelief and organized working 
committees, produced research tools and began their investigations. Although results so far 
have been incomplete and inconclusive, some of them arc summarized below. 



Plaiming 



Topics 



Questionnaires 



Personnel 



Reporting 
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Group Work 

i Teachers are concerned that the noise level in their classes during groupwork may 
*upset' their Principals. 

ii The good students contribute most of the work during groupwork; the weaker 
students remain silent. 

ill There is a tendency during groupwork for students to use Bahasa Malaysia or 
their mother tongue. 



Integra tion_of_Skills 

i Teachers feel that more attention should first be given to oral skills. 

ii Students are reluctant to speak. 

iii There is insu^cient time in the class to integrate all language skills. 



Moral Values 

i It is di£ficult to include moral values in all lessons. 

ii Sometimes lessons may become boring because of attempts to incorporate moral 
values. 

What wc have learned at the Ministry is: 
Rome was not built in a day 

In the interests of having a firm base of experienced and trained personnel we are 
prepared to allow time for projects to be trialled, refined and tried again. We think that this 
will allow our officers to gain expertise from personal experience. 

More haste less speed. 

We regard the formative evaluation work as a long term, ongoing part of the 
administration of our education system. We believe that rushing things at the beginning of 
the project, and we feel that we are still at the beginning of our work, will only create 
problems for us at a later date. 

We have tried to ^ve an overview of our formative evaluation project which, aimmg 
towards a national level investigation, naturally involves many people from different areas of 
our education network. Perhaps the most important thing we have learned is that evaluation 
of educational processes is, like education itself, continuous and long term. 
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Appendix I: Project Stages 



The stages of our project are: 



STAGE 



CONTENT 



phase one 
Mclaka Seminar 
January 1989 



Introduce personnel to the aims and 
benefits of formative evaluation. 
Demonstrate types of data collection. 
Instigate first attempts. 



phase two 
Penang Seminar 
August 1989 



Review first attempts. 

Introduce a quarterly newsletter. 

Schools Division to visit each state to monitor 

work and Tme tune' instruments. 



phase three 
Genting Seminar 

July 1990 



Review 

Selection of state level instruments for national 
level work. 

of existing instruments. 

Framework of channels of reporting from school 
level via state level 
to ministry level. 

Prepare administrative network (e.g. job 
descriptions). 

Introduce personnel to quantitative procedures, 
and train personnel in more sophisticated ways 
of analysis and reporting. 
Instigate National level investigations. 



phase four 
December 1990 



Review of progress and available instruments. 
Seminar and workshops to introduce formative 
evaluation work into schools nationally. 
Schools Division to visit each state to help with 
data collection 



phase five 



Interpretation of first set of March 1991 
fmdings by Schools Division. 
Preparation of report on initial national 
formative evaluation. 

Survey and related recommendations for 

submission to Ministry 

officials. 
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Appendix II: Areas of investigation 



Focusing on Methodolopv 

teachers' methodology 



classroom management 
managing group activities 

managing pair work 
integration of skills & content 



use of teaching aids and 
materials 



questionnaires 
(teachers & students) 
class observation 
interviews 
pro-forms 

observations 
questionnaires 

questionnaires 
interviews 

(students & teachers) 

questionnaires 
interviews 

questionnaires 
interviews 

questionnaires 
checklists 
interviews 
(teachers) 



investigation of teaching and learn- questionnaires 
learning techniques in the KBSM syllabus 



a study of non-optionists 
(ELT teachers whose main 
discipline is not English) 



observation 

self-evaluation 

forms. 

questionnaires 
interviews 
diary studies 



Appendix III : Areas of mvestigation 



Focusing on Learning 

class participation/student 
interaction 



questionnaires and 
interviews 

(students & teachers) 



story telling & language games questionnaires 

(students) 



students' work activities 

the level of class participation 



student 

questionnaires 

questionnaires and 
interviews 

(teachers & students) 
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Appendix IV: State Formative Evaluation Projects 



Formative Evaluation projects are being conducted m each state in Malaysia. 



State 



Pedis 



Personnel 



2JPN officers 
8 teachers 
questionnaires 



Number of 
schools 

16 secondary 
schools 



Area of 
investigation 

teaching 
techniques 



Data collection 
technique(s) 

questionnaires 
(students 

(teachers) 

lesson 

observations 



Kedah 



Pulau 
Pinang 



2 JPN officers 

3 teachers 



2 JPN officers 
1 Key Personnel 
1 teacher 



30 secondary 
schools 



20 secondary 
schools 



students 
homework 



students' 
class 

(10 urban) 
(10 rural) 



questionnaires 
(forms I & II 
assignments 
students) 

questionnaires 
interviews 
participation 
students class 
interaction 



Perak 



3 JPN officers 

12 Resource 

Persons 

9 Assistant 

District 

Education 

officers 



20 secondary 



teachers' 



schools of teaching checklist 
aids and 
teaching 
materials 



use 



questionnaires 



Kelantan 3 JPN officers 
20 teachers 



20 secondary non-optionists 
schools using KBSM 



lesson 

observations 

checklists 

self-evaluation 

forms 

questionnaires 
in reviews 
diary studies 



Terengganu 1 JPN officer 
teachers 



11 secondary 
schools 



teaching 
methodologies 



questionnaires 
interviews 
10 State and 
District 



Pahang 2 JPN officers 15 secondary 
12 Key Personnel schools 



teaching 
methodologies 
in KBSM 
learning 
opportunities 
in KBSM 



questionnaires 
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State Personnel 



Number of 
schools 



Area of 
InvestigEtion 



Data collection 
technique(s) 



Wllayah 3 JPN officers 
Persekutuan2 Key Persons 
Supervisor 
3 teachers 



15 secondary 
schools 



story telling 
and language 



questionnadres 
1 English games 



Selangor 



Johor 



Negcri 
Sembilan 

Melaka 



2 JPN officers 

3 Resource 
Persons 

1 Language 
Supervisor 
teachers 
students 

1 JPN officer 
17 Key Persons 



4 JPN officers 

3 Resource 

4 JPN officers 
3 Resource 

Persons 



15 secondary 
schools 



pairwork 
activities 



questionnaires 
interviews 



142 secondary 
schools 

(both rural 
and urban) 

10 secondary 
schools 

12 secondary 
schools 



groupwork 
activities 



classroom 
management 

integration 
of skills & 

content 



questionnaires 

lesson 

observations 



observations 
questionnaires 

questionnaires 
(students & 
teachers) 
interviews 
small group 
(students and 
activities 
teachers) 
moral values 



Sarawak 



2 JPN officers 
1 Assistant 
Principal 
4 teachers 



12 secondary 
schools 



managing 

group 

activities 



questionnaires 
interviews 



Sabah 



2 JPN officers 

School 

Inspectors 

2^ne Heads 

Principals 

Resource 

Persons 

Teachers 



10 secondary 
schools 



teaching 
methodology 



questionnaires 

interviews 

proforma 
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